Leveraging machine learning and fractal analysis for classifying motion

ABSTRACT

A machine learning and fractal analysis process for classifying human or animal motion, including the classification of patterns generated by human or animal motion in order to assess the quality of athletic performance, artistic performance, form, or other quality of motion.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/837,188 filed on Apr. 1, 2020; which claims the benefit of U.S.Provisional Application Ser. No. 62/828,152 filed on Apr. 2, 2019, thecontents of which are all expressly incorporated herein by reference intheir entirety.

BACKGROUND

The statements in this background section merely provide backgroundinformation related to the present disclosure and should not beconstrued as constituting prior art.

Authenticating abstract drip painting as being authentic paintings ofthe original master artist is a challenge due to the nature of thepieces. To create an artwork in this style, a canvas is placed on thefloor, and the artist allows paint to drip down from the brush to thecanvas. As the artist moves over the canvas, paint is dropped from thebrush creating a trace of the movement patterns made by the painter.

Additionally, every individual's body has a set of bio-mechanical motionvariations (bio-variations), or natural motions, which interact atmultiple scales to drive many forms of activity from dance to thecreation of art. These bio-variations exhibit subtle variations fromindividual to individual (e.g., elite athletes vs. casual joggers,famous artists vs crude imitators, etc.), but also within individualswhen their condition changes (e.g., from fatigue, physical injury,psychological, physiological or neurological conditions, age,concentration, etc.).

One bio-variation pattern of particular interest is the identificationof fatigue in running. In particular, running related injuries are verycommon, and it is believed that a major cause of such injuries is due tothe breakdown of running form once the runner becomes fatigued. To thisend, it is also suspected that many running related injuries are alsocaused by poor running form in response to muscle weakness or poor motorcontrol patterns.

In a physical therapy clinic, a therapist works with a patient to guideexercises and ensure that the patient is performing the movements withproper form. Patients who are not familiar with the exercises are oftenhesitant to practice on their own for fear of injury, or mayaccidentally injure themselves by performing the exercise improperly.

The technologies described herein are intended to address and/or toimprove upon some of the above-mentioned issues and challenges withinthe relevant industry.

SUMMARY

The technologies described herein are concerned, for example, withassessing the quality of movement patterns, whether artistic, athletic,or otherwise. Although conventional systems may record the quantity ofmovement, such as the number of steps taken, distance traveled, ornumber of repetitions executed, or may provide feedback concerning alimited number of key measurements, such systems do not assess thequality of the entire pattern. To accomplish this, the technologiesdescribed herein incorporate fractal analysis techniques (e.g., inconjunction with machine learning techniques) in the quantification ofhuman or animal movement patterns.

Moreover, although conventional systems may be capable of presenting atime series graph of measurements (e.g., acceleration vs. time), suchsystems are incapable of generating an aesthetic orbital diagram thatallows the pattern to be viewed in a more intuitive manner by the user.More particularly, the technologies described herein allow the creationof a 2D projection of motion, which in turn allows for the use ofquicker, more reliable, and more advanced machine vision algorithms.

The technologies described herein may also involve a calibrationprocedure that may be used as a baseline when detecting running fatiguein a unique manner as described herein. More particularly, thetechnologies described herein may utilize a calibration procedure thattrains a machine learning algorithm to assess movement quality with theassistance of a skilled observer. After the movement quality has beenassessed, the system records the captured movement observations within adatabase that includes patterns of individuals who have been assessed bya qualified third party, such as, for instance, a licensed physicaltherapist, a health care practitioner or a coach. This data can then beused to train a machine learning algorithm to classify various movementpatterns.

In some embodiments, the technologies described herein include thecollection of movement and/or bio-variation data from users performingvarious activities, and a unique gestural art visualization may becreated based on such data. The created visualization, in turn,highlights the subtle variations in the movement pattern, which can thenbe further analyzed and categorized, for example, using fractal analysisand machine learning algorithms.

In some embodiments, the technologies described herein may furtherprovide a method for determining when a runner's form begins to exhibitmovement patterns associated with fatigue. Further, the detectedvariations in movement pattern may be relayed to the runner as a meansto recommend when the user should take a break and rest in order toavoid potential injury. Moreover, in some embodiments, the detectedmovement pattern variations of the runner may also be compared withclinically diagnosed individuals exhibiting similar dysfunctions in aneffort to further diagnose improper form, and thus provide additionalcorrective feedback.

It should be appreciated that, in some embodiments, a diagnostic toolutilizing movement sensors may be available to physical therapists,trainers, and coaches for the purpose of training a user in conjunctionwith a machine learning algorithm. For example, after the system hasbeen trained to identify proper form in exercise (with guidance from aphysical therapist or coach), the user can use the system independentlyand receive feedback based on the quality of the user's movements. Thismay allow the patient to exercise outside of the clinic with a reducedlikelihood of injury due to improper form. Additionally, in someembodiments, the system can count sets and repetitions performed by theuser, thereby allowing both patient and therapist to comply with anexercise program.

In some embodiments, the technologies described herein may provide foranalyzing movement patterns (e.g., other than running or physicaltherapy) for a variety of exercises in which improper movements may leadto injury or where feedback regarding movement quality may lead toimproved performance. For example, the technologies described herein maybe used in conjunction with weight lifting, baseball pitching, golfswinging, martial arts, dancing, and/or other movements/activities.

In some embodiments, the technologies described herein provide foranalyzing the movement patterns of patients suffering fromneurodegenerative diseases, such as Parkinson's disease, Alzheimer'sdisease, Huntington's disease, ALS, and/or other conditions. It has beenshown that exercise can be beneficial in the treatment of manyneurodegenerative diseases; however, there is an increased risk ofinjury due to falling or poor movement mechanics. Accordingly, thetechnologies described herein may be used to monitor movement patternsand provide feedback for these patients as a means to minimize the riskof injury.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used as an aid inlimiting the scope of the claimed subject matter. Further embodiments,forms, features, and aspects of the present application shall becomeapparent from the description and figures provided herewith.

BRIEF DESCRIPTION OF THE DRAWINGS

The concepts described herein are illustrative by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. Where considered appropriate, referenceslabels have been repeated among the figures to indicate corresponding oranalogous elements.

FIG. 1 is a simplified block diagram of at least one embodiment of asystem for leveraging machine learning and/or fractal analysis toclassify motion;

FIG. 2 is a simplified flow diagram of at least one embodiment of amethod for leveraging machine learning and/or fractal analysis toclassify motion using the system of FIG. 1 ;

FIG. 3 depicts the projection of three-dimensional human motion inacceleration space to two-dimensional images, which can be analyzed bymachine vision algorithms in accordance with the techniques describedherein;

FIG. 4 depicts a resulting pattern of three-dimensional human motion inacceleration space projected to a two-dimensional still image inaccordance with the techniques described herein; and

FIG. 5 depicts a two-dimensional projection of three-dimensional humanmotion created by dripping paint onto a canvas on the floor, such thatthe resulting pattern can be analyzed using machine vision and fractalanalysis algorithms in order to authenticate original paintings inaccordance with the techniques described herein.

DETAILED DESCRIPTION

Although the concepts of the present disclosure are susceptible tovarious modifications and alternative forms, specific embodiments havebeen shown by way of example in the drawings and will be describedherein in detail. It should be understood, however, that there is nointent to limit the concepts of the present disclosure to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives consistent with the presentdisclosure and the appended claims.

References in the specification to “one embodiment,” “an embodiment,”“an illustrative embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may or may not necessarily includethat particular feature, structure, or characteristic. Moreover, suchphrases are not necessarily referring to the same embodiment. It shouldfurther be appreciated that although reference to a “preferred”component or feature may indicate the desirability of a particularcomponent or feature with respect to an embodiment, the disclosure isnot so limiting with respect to other embodiments, which may omit such acomponent or feature. Further, when a particular feature, structure, orcharacteristic is described in connection with an embodiment, it issubmitted that it is within the knowledge of one skilled in the art toimplement such feature, structure, or characteristic in connection withother embodiments whether or not explicitly described. Additionally, itshould be appreciated that items included in a list in the form of “atleast one of A, B, and C” can mean (A); (B); (C); (A and B); (B and C);(A and C); or (A, B, and C). Similarly, items listed in the form of “atleast one of A, B, or C” can mean (A); (B); (C); (A and B); (B and C);(A and C); or (A, B, and C). Further, with respect to the claims, theuse of words and phrases such as “a,” “an,” “at least one,” and/or “atleast one portion” should not be interpreted so as to be limiting toonly one such element unless specifically stated to the contrary, andthe use of phrases such as “at least a portion” and/or “a portion”should be interpreted as encompassing both embodiments including only aportion of such element and embodiments including the entirety of suchelement unless specifically stated to the contrary.

The disclosed embodiments may, in some cases, be implemented inhardware, firmware, software, or a combination thereof. The disclosedembodiments may also be implemented as instructions carried by or storedon one or more transitory or non-transitory machine-readable (e.g.,computer-readable) storage media, which may be read and executed by oneor more processors. A machine-readable storage medium may be embodied asany storage device, mechanism, or other physical structure for storingor transmitting information in a form readable by a machine (e.g., avolatile or non-volatile memory, a media disc, or other media device).

In the drawings, some structural or method features may be shown inspecific arrangements and/or orderings. However, it should beappreciated that such specific arrangements and/or orderings may not berequired. Rather, in some embodiments, such features may be arranged ina different manner and/or order than shown in the illustrative figuresunless indicated to the contrary. Additionally, the inclusion of astructural or method feature in a particular figure is not meant toimply that such feature is required in all embodiments and, in someembodiments, may not be included or may be combined with other features.

The terms longitudinal, lateral, and transverse may be used to denotemotion or spacing along three mutually perpendicular axes, wherein eachof the axes defines two opposite directions. The directions defined byeach axis may also be referred to as positive and negative directions.Additionally, the descriptions that follow may refer to the directionsdefined by the axes with specific reference to the orientationsillustrated in the figures. For example, the directions may be referredto as distal/proximal, left/right, and/or up/down. It should beappreciated that such terms may be used simply for ease and convenienceof description and, therefore, used without limiting the orientation ofthe system with respect to the environment unless stated expressly tothe contrary. For example, descriptions that reference a longitudinaldirection may be equally applicable to a vertical direction, ahorizontal direction, or an off-axis orientation with respect to theenvironment. Furthermore, motion or spacing along a direction defined byone of the axes need not preclude motion or spacing along a directiondefined by another of the axes. For example, elements described as being“laterally offset” from one another may also be offset in thelongitudinal and/or transverse directions, or may be aligned in thelongitudinal and/or transverse directions. The terms are therefore notto be construed as further limiting the scope of the subject matterdescribed herein.

As described in greater detail below, the technologies described hereinallow for the collection of bio-variation data from users performingvarious activities. To do so, input data may be gathered from a varietyof sources including, but not limited to, paint traces of human oranimal movement, inertial sensors (e.g., an IMU) relaying accelerationor orientation data, video, still images showing movement traces, andother data. In some embodiments, a method of analysis described hereininvolves a still image showing the trajectory of a repetitive movementpattern. For instance, inertial sensor data can be recorded andprocessed by projecting a three-dimensional movement pattern into atwo-dimensional plane. Such a projection may essentially be a digitalversion of dropped paint onto a canvas that has been analyzed for aresulting pattern. If data is recorded in higher dimensions, it mayfirst be projected to a two-dimensional trace prior to analysis (e.g.,(x,y,z) (x,y), (x,z), (y,z)). Data points may then joined with a line orcurve in order to create a two-dimensional image that can be inputtedinto the machine learning process. Instead of performing a traditionaltime series analysis of the motion sample, two-dimensional images may becreated from each plane (e.g., side, front, top) and may be analyzed asa static image similar to an abstract drip painting. It should beappreciated that conversion to a two-dimensional image allows for theuse of highly efficient and optimized machine vision algorithms.Additionally, a visually aesthetic image may be created by overlayingmultiple repetitions of the movement pattern in question, and this imagecan be presented to the user for intuitive feedback. It should beappreciated that the various motions can be displayed in x, y, z spaceto generate a “spatial portrait” of the individual or displayed inacceleration space (a_(x), a_(y), a_(z)) to generate a “force portrait”of the individual. In other words, the various projections describedherein may be made from higher dimensional data sets in positionalspace, acceleration space, and/or other suitable spaces. For example, insome embodiments, the projections may be from momentum space, velocityspace, and/or other spaces. Further, it should be appreciated that thehigher dimensional data sets from which the projections are made may beassociated with non-inertial sensor data in other embodiments. Dependingon the particular embodiment, the various “portraits” or projections maybe used separately or together to capture the individual's uniquemotions. In some embodiments, data can also be sampled and transmittedto a mobile device, computer, or cloud for further analysis andfeedback.

In embodiments in which data is acquired via inertial sensors, thesensors may be first placed in specific locations on the user's body. Insome embodiments, the sampling rates for data acquisition may beapproximately 1 kHz; however, it should be appreciated that the samplingrates may vary in other embodiments. Similarly, accelerometersensitivity may also vary, for example, depending on the sensorplacement or specific application. It should be appreciated that variousdifferent communication technologies may be utilized depending on theparticular embodiment including, for example, wireless transmission to aremote device (e.g., a mobile device or computer) via Bluetooth, Wi-Fi,or communication technology. In addition, in some embodiments, data maybe recorded directly to the sensor device (and/or another computingdevice) to be processed and analyzed at a later time. Moreover,depending on the particular embodiment, sensors may transmit dataindividually, or multiple sensors may be wired together and all sensordata may be transmitted from a single transmitter.

As described herein, after the data has been acquired and processed, itmay be classified using a combination of machine learning and fractalanalysis techniques. For example, in some embodiments, the variousmovement patterns of the user may be classified using such techniques.Further, in some embodiments, in addition to machine learningclassification and fractal analysis of the movement patterns, anaesthetic image of the repetitive movement trajectory may also begenerated from the inertial sensor data. In some embodiments, thegenerated image may be presented to the user as feedback and/or to trackprogress over time by either demonstrating improvement with practice orreduction of movement quality with fatigue. Images may also be sharedwith coaches, physical therapists, trainers, and/or other users.

It should be understood and appreciated herein that the technologiesdescribed herein have several features that are advantageous,particularly when compared to other processes known within the art. Forexample, in some embodiments, the technologies described herein make useof advanced machine vision algorithms by analyzing a two-dimensionalprojection of higher dimensional orbit as opposed to traditional timeseries analysis. Such a technique is particularly useful because machinevision algorithms developed mainly for use in other fields are very fastand accurate. Unlike conventional systems that require video for suchmachine learning techniques, the techniques described herein allow forthe use of inertial sensors, paint, and/or other measurement devices.

In some embodiments, the technologies described herein may be leveragesto measure fatigue in running specific to the individual beingmonitored. Other systems, by contrast, use a broad “one size fits all”assessment of biomechanics. Further, in other embodiments, thetechnologies described herein allow for the calibration of exerciseroutines to be guided by a physical therapist or experienced coach. Thisis particularly useful because the associated exercises are guided andindividualized, and the machine learning algorithm may be therebytrained as the user is trained by the skilled observer. Other examplesand embodiments are further described in detail below.

Referring now to FIG. 1 , in the illustrative embodiment, a system 100for leveraging machine learning and/or fractal analysis to classifymotion includes a wearable computing system 102, a network 104, and aserver 106. The wearable computing system 102 may be embodied as anytype of computing device capable of being worn by a user (e.g., a smartwatch, sensor assembly, cellular phone, smartphone, etc.). In otherembodiments, it should be appreciated that the wearable computing system102 may instead be embodied as one or more separate (i.e., non-worn)devices communicatively coupled to one or more sensors (e.g., worn by auser). As such, in various embodiments, it should be appreciated thatthe wearable computing system 102 may be embodied as and/or include awearable computing device, a desktop computer, laptop computer, tabletcomputer, notebook, netbook, Ultrabook™, cellular phone, smartphone,personal digital assistant, mobile Internet device, Internet of Things(IoT) device, server, router, switch, and/or any othercomputing/communication device capable of performing the functionsdescribed herein.

As shown in FIG. 1 , the illustrative wearable computing system 102includes a processor 110, an input/output (“I/O”) subsystem 112, amemory 114, data storage 116, a communication circuitry 118, and one ormore sensors 120. Of course, the wearable computing system 102 mayinclude other or additional components, such as those commonly found ina typical computing device/system (e.g., various input/output devicesand/or other components), in other embodiments. Additionally, in someembodiments, one or more of the illustrative components, or a portionthereof, may be incorporated in the processor 110 in some embodiments.Although a single wearable computing system 102 is illustratively shown,it should be appreciated that one or more of the components of thewearable computing system 102 described herein may be distributed acrossmultiple computing devices. In other words, the techniques describedherein may be employed by a computing system that includes one or morecomputing devices.

The processor 110 may be embodied as any type of processor capable ofperforming the functions described herein. For example, the processor110 may be embodied as a single or multi-core processor(s), digitalsignal processor, microcontroller, or other processor orprocessing/controlling circuit. Similarly, the memory 114 may beembodied as any type of volatile or non-volatile memory or data storagecapable of performing the functions described herein. In operation, thememory 114 may store various data and software used during operation ofthe wearable computing system 102 such as operating systems,applications, programs, libraries, and drivers. The memory 114 iscommunicatively coupled to the processor 110 via the I/O subsystem 112,which may be embodied as circuitry and/or components to facilitateinput/output operations with the processor 110, the memory 114, andother components of the wearable computing system 102. For example, theI/O subsystem 112 may be embodied as, or otherwise include, memorycontroller hubs, input/output control hubs, firmware devices,communication links (i.e., point-to-point links, bus links, wires,cables, light guides, printed circuit board traces, etc.) and/or othercomponents and subsystems to facilitate the input/output operations. Insome embodiments, the I/O subsystem 112 may form a portion of asystem-on-a-chip (SoC) and be incorporated, along with the processor110, the memory 114, and other components of the wearable computingsystem 102, on a single integrated circuit chip. For example, in someembodiments, one or more of the components of the wearable computingsystem 102 may form one or more application-specific integrated circuits(ASICs).

The data storage 116 may be embodied as any type of device or devicesconfigured for short-term or long-term storage of data such as, forexample, memory devices and circuits, memory cards, hard disk drives,solid-state drives, or other data storage devices. The data storage 116and/or the memory 114 may store various data during operation of thewearable computing system 102 useful for performing the functionsdescribed herein.

The communication circuitry 118 may be embodied as any communicationcircuit, device, or collection thereof, capable of enablingcommunications between the wearable computing system 102 and otherremote devices (e.g., the server 106) over a network (e.g., the network104). The communication circuitry 118 may be configured to use any oneor more communication technologies (e.g., wireless or wiredcommunications) and associated protocols (e.g., Ethernet, Bluetooth®,Wi-Fi®, WiMAX, etc.) to bring about such communication.

Each of the sensors 120 is configured to generate sensor data (e.g., byvirtue of one or more signals), which may be interpreted by theprocessor 110 to determine one or more characteristics associated withthe environment thereof. By way of example, the sensors 120 may detectvarious physical characteristics, electrical characteristics, and/orelectromagnetic characteristics of its environment. For example, in theillustrative embodiment, one or more of the sensors 120 may be embodiedas, or otherwise include, at least one inertial sensor (e.g.,accelerometer, gyroscope, etc.). In other embodiments, the sensors 120may include one or more other inertial sensors, environmental sensors,proximity sensors, optical sensors, electromagnetic sensors (e.g.,magnetometers), audio sensors, motion sensors, piezoelectric sensors,cameras, and/or other types of sensors. Further, the wearable computingsystem 102 may also include components and/or devices configured tofacilitate the use of the sensors 120. In some embodiments, it should beappreciated that multiple sensors 120 may be included in a single sensormodule (e.g., an inertial measurement unit).

Although the sensors 120 are depicted as forming a portion of thewearable computing system 102 in FIG. 1 , it should be appreciated thatone or more of the sensors 120 may be separate from the wearablecomputing system 102 in some embodiments. Further, in such embodiments,the sensors 120 may be communicatively coupled to the wearable computingsystem 102 via a suitable wired or wireless communication connection. Insome embodiments, each of such distributed sensors 120 may be configuredto communicate directly with the wearable computing system 102, whereasin other embodiments the distributed sensors 120 may include anaggregating sensor 120 configured to collect the sensor data generatedby one or more other distributed sensors 120 for transmission to thewearable computing system 102 (e.g., via a single communication path).

Further, in some embodiments, the wearable computing system 102 may beconfigured to be coupled to one or more peripheral devices. It should beappreciated that the particular peripheral devices may include anynumber of additional peripheral or interface devices, such as speakers,microphones, additional storage devices, and so forth, and may dependon, for example, the type and/or intended use of the wearable computingsystem 102. For example, in some embodiments, the peripheral devices mayinclude a keyboard, mouse, display, touchscreen display, printer, alarm,status indicator, handheld device, diagnostic tool, and/or one or moreother suitable peripheral devices.

The network 104 may be embodied as any type of communication networkcapable of facilitating communication between the wearable computingsystem 102 and remote devices (e.g., the server 106). As such, thenetwork 104 may include one or more networks, routers, switches,computers, and/or other intervening devices. For example, the network104 may be embodied as or otherwise include one or more cellularnetworks, telephone networks, local or wide area networks, publiclyavailable global networks (e.g., the Internet), ad hoc networks,short-range communication links, or a combination thereof.

The server 106 may be embodied as any type of computing device capableof performing the functions described herein. For example, the server106 may be embodied as a server, desktop computer, laptop computer,tablet computer, notebook, netbook, Ultrabook™ cellular phone,smartphone, wearable computing device, personal digital assistant,mobile Internet device, Internet of Things (IoT) device, router, switch,and/or any other computing/communication device capable of performingthe functions described herein. In some embodiments, the server 106 mayinclude components similar to the components of the wearable computingsystem 102 described above and, therefore, the descriptions of thosecomponents have not been repeated herein for clarity of the description.Further, it should be appreciated that the server 106 may include othercomponents, sub-components, and/or devices commonly found in a computingdevice, which are not discussed herein for clarity of the description.Additionally, in some embodiments, one or more of the components of thewearable computing system 102 may be omitted from the server 106 (e.g.,the sensors 120).

Although only one wearable computing system 102, one network 104, andone server 106 are shown in the illustrative embodiment of FIG. 1 , thesystem 100 may include multiple wearable computing systems 102, networks104, and/or servers 106 in other embodiments. For example, in someembodiments, the server 106 may communicate with multiple wearablecomputing systems 102. Further, in some embodiments, it should beappreciated that the wearable computing system 102 may perform all ofthe functions described herein (e.g., the functions of both the wearablecomputing system 102 and the server 106). In such embodiments, thenetwork 104 and the server 106 may be omitted from the system 100.

It should be further appreciated that, although the server 106 isdescribed herein as a device and/or system outside of a cloud computingenvironment, in other embodiments, the server 106 may be embodied as orinclude a cloud-based device or collection of devices within a cloudcomputing environment. Further, in cloud-based embodiments, the server106 may be embodied as a server-ambiguous computing solution, forexample, that executes a plurality of instructions on-demand, containslogic to execute instructions only when prompted by a particularactivity/trigger, and does not consume computing resources when not inuse. That is, the server 106 may be embodied as a virtual computingenvironment residing “on” a computing system (e.g., a distributednetwork of devices) in which various virtual functions (e.g., Lambdafunctions, Azure functions, Google cloud functions, and/or othersuitable virtual functions) may be executed corresponding with thefunctions of the server 106 described herein. For example, when an eventoccurs (e.g., data is transferred to the processor for handling), thevirtual computing environment may be communicated with (e.g., via arequest to an API of the virtual computing environment), whereby the APImay route the request to the correct virtual function (e.g., aparticular server-ambiguous computing resource) based on a set of rules.As such, when a request for the transmission of certain data is made(e.g., via an appropriate user interface to the server 106), theappropriate virtual function(s) may be executed to perform the actionsbefore eliminating the instance of the virtual function(s).

Referring now to FIG. 2 , in use, the system 100 (e.g., in conjunctionwith one or more users) may execute a method 200 for leveraging machinelearning and/or fractal analysis to classify motion. It should beappreciated that the features described in reference to the method 200may be performed in conjunction with the various methods and/or examplesdescribed herein. For example, the method 200 may be executed forauthenticating artwork, determining whether a user's motion isindicative of fatigue, determining whether the user's motion is inproper form, and/or for other suitable purposes. It should be furtherappreciated that the particular blocks of the method 200 are illustratedby way of example, and such blocks may be combined or divided, added orremoved, and/or reordered in whole or in part depending on theparticular embodiment, unless stated to the contrary.

The illustrative method 200 begins with block 202 in which at least onemachine learning algorithm is trained (or training of such an algorithmbegins). Depending on the particular machine learning algorithm usedand/or the particular embodiment, it should be appreciated that themachine learning algorithm may be supervised or unsupervised. Further,in some embodiments, the user does not need to perform any calibrationof the sensors and/or the machine learning algorithm. The machinelearning algorithm(s) leveraged by the system 100 may include one ormore neural network algorithms, regression algorithms, instance-basedalgorithms, regularization algorithms, decision tree algorithms,Bayesian algorithms, clustering algorithms, association rule learningalgorithms, deep learning algorithms, dimensionality reductionalgorithms, and/or other suitable machine learning algorithms,techniques, and/or mechanisms.

In some embodiments, the machine learning algorithm may be trained usinga large library of movement samples collected in the lab or clinic. Forexample, the samples used for training may be recorded from manydifferent individuals, some of whom may exhibit movement dysfunctionsthat have been diagnosed by a licensed physical therapist, while othersmay be from individuals possessing exemplary movement mechanics. Inparticular, inertial sensor data may be recorded and processed in amanner similar to that described below. As such, a database may bedeveloped by recording the movement patterns of individuals who havebeen clinically assessed by a licensed physical therapist, trainer, orother professional based on the particular motion being assessed.Furthermore, many different movement patterns may be classified andsamples recorded from many individuals, which collectively exhibit arange in the quality of movement mechanics. In some embodiments, samplesmay be recorded for a variety of movements and exercises includingrunning, lifting weights, and other exercises where proper movementtechnique is critical. These patterns may then be used to train amachine learning algorithm in order to provide feedback to the user, aswell as may be used by a coach or physical therapist to help diagnosemovement dysfunctions in the user.

In block 204, the wearable computing system 102 records sensor datagenerated by the sensors 120. As discussed above, in some embodiments,the sensors 120 may include inertial sensors in some embodiments, whichmay be positioned on suitable positions on the user's body depending onthe particular movements being assessed.

In block 206, the wearable computing system 102 projects athree-dimensional movement pattern indicated by the sensor data to oneor more two-dimensional planes to generate corresponding two-dimensionalprojections. For example, in some embodiments, the three-dimensionalmovement pattern indicated by the sensor data may be projected ontomultiple two-dimensional planes (e.g., a side plane, front plane, andtop plane).

In block 208, the wearable computing system 102 analyzes thetwo-dimensional projection(s). In doing so, it should be appreciatedthat the wearable computing system 102 may leverage computer vision inblock 212, machine learning in block 214 (e.g., the algorithms trainedin block 202), and/or fractal analysis in block 216. As indicated above,in some embodiments, the two-dimensional projections, sensor data,and/or other data may be transmitted to a remote computing device inorder to perform such analysis (e.g., in full or in part).

The wearable computing system 102 and/or the system 100 may applyvarious computer vision algorithms, filters, and/or techniques togenerate processed versions of the two-dimensional projections and/orreformatted versions thereof. For example, in some embodiments, thewearable computing system 102 and/or the system 100 may utilize one ormore image filters (e.g., kernel-based convolution, masking, etc.), edgedetection algorithms (e.g., Canny edge detection, Sobel filters, etc.),image segmentation algorithms (e.g., pyramid segmentation, watershedsegmentation, etc.), blob detection algorithms, corner detectionalgorithms, features identification and/or matching algorithms (e.g.,scale-invariant feature transform (SIFT), speeded-up robust features(SURF), etc.), morphological image processing algorithms (e.g., erosion,dilation, opening, closing, etc.), threshold/voting algorithms, and/orother suitable algorithms useful in analyzing the two-dimensionalprojections.

As indicated above, the particular machine learning algorithm(s)leveraged by the wearable computing system 102 and/or the system 100 mayvary depending on the particular embodiment. For example, in variousembodiments, the wearable computing system 102 and/or the system 100 mayutilize one or more neural network algorithms, regression algorithms,instance-based algorithms, regularization algorithms, decision treealgorithms, Bayesian algorithms, clustering algorithms, association rulelearning algorithms, deep learning algorithms, dimensionality reductionalgorithms, and/or other suitable machine learning algorithms,techniques, and/or mechanisms in analyzing the two-dimensionalprojections.

It should be appreciated that fractal analysis proceeds by dividing thepattern into successively smaller grid sizes, counting the number ofsquares in the grid filled at a given scale, and plotting the results ona log-log graph. Standard fractals will yield a straight line indicatingscale invariance. The slope of this line is a measure of the complexityof the movement pattern. Some patterns may exhibit multi-scaledeviations from a perfectly straight-line fit, and these variations maybe useful in determining unique characteristics of the pattern inaddition to the slope of the linear fit indicating complexity. There areseveral variations of fractal analysis that may be employed, including,but not limited to, spatial analysis, temporal fractal analysis,information dimension, multi-fractal analysis, and others. It should beappreciated that fractal analysis may be leveraged in conjunction withmachine learning to provide a unique assessment of the movement pattern.

Unlike other techniques, the techniques described herein may use amulti-scale analysis, which takes into account variations in the fractaldimension at different scales. Further, merging machine learning withfractal analysis allows the system 100 to more completely describe thescaling properties of the bio-variation pattern being analyzed. Atypical fractal analysis, by contrast, performs a linear regression onthe scaling data in a log-log plot. The techniques described herein,however, may utilize machine learning to find a more complex curve tofit the scaling data, which provides a more comprehensive measurement ofthe subtle variations in the movement pattern.

As described in further detail throughout, it should be appreciated thatthe analysis of the projections may serve various purposes depending onthe particular context of the analysis and/or the particular embodiment.For example, in various examples, the analysis of the projections mayinclude authenticating artwork generated by the user based on the user'smovement, determining whether the user's movement is indicative offatigue, determining whether the user's movement corresponds with aproper movement form, and/or analyzing the projections to determineother relevant characteristics useful in providing feedback regardingthe user's movement.

In block 218, the wearable computing system 102 provides feedback to theuser and/or another party regarding the user's movement based on theanalysis of the projection(s). As part of such feedback, or separate tosuch feedback, in block 220, the wearable computing system 102 generatesone or more images indicative of the user's movement based on theanalysis of the projection(s). In some embodiments, the image may begenerated by overlaying multiple repetitions of the movement patternand/or generating a “heat map” indicative of the user's movement. Theimage generated by the wearable computing system 102 may be referred toherein as a “portrait.”

Although the blocks 202-220 are described in a relatively serial manner,it should be appreciated that various blocks of the method 200 may beperformed in parallel in some embodiments.

Various processes and methods of classifying patterns generated by humanmotion in accordance with the illustrative teachings of the presentdisclosure are demonstrated in the following examples. These examplesare illustrative only and are not intended to limit or preclude otherembodiments. For instance, it should be understood and appreciatedherein that the teachings of the present disclosure may also be used toclassify other patterns of human motion. Additionally, in someembodiments, the technologies described herein may be used to analyzepatterns of non-human animal motion (e.g., the gait/trot/gallop of ahorse).

Example 1: Abstract Art Authentication

With a canvas placed on the floor, an artist may perform movements overthe canvas and allow paint to drop from the brush down to the floor.This technique essentially traces the artist's three-dimensionalmovement patterns and projects them onto a two-dimensional plane. Inthis way, a unique pattern is created, which captures subtle movementsof the artist. Other forms of gestural abstract art may also be assessedin accordance with such techniques method (e.g., essentially any artworkproduced by large-scale motions involving full body movements). In someembodiments, fractal analysis is performed in order to quantify thecomplexity of the movement pattern and to determine the uniquemulti-scale bio-variation fractal “portrait” of the artist. Moreover,the fractal analysis may measure the degree to which the movementpattern adheres to fractal scale-invariance. The unique fractal“portrait” can then be used to distinguish artworks and individualmovement patterns.

A machine learning algorithm may be trained with examples of an artist,and subsequent paintings can be classified by comparison with theoriginal training set in order to determine the authenticity of theartwork. Machine learning may also be employed to visualize the featurespace of a selection of artworks as a representation of how similarthese artworks are to one another.

Example 2: Detecting Fatigue in Running

Inertial sensors may be placed on an individual such that data can besampled as the user runs. It should be understood and appreciated hereinthat sensors may be placed anywhere on the body depending on whatpatterns are to be sampled. To this end, in some embodiments, placementon the sacrum may be particularly useful for identifying movementdysfunctions and fatigue.

A primary objective of this technique is to identify fatigue, therebyminimizing the risk of injury. To this end, it should be understood thatfatigue may be exhibited in bio-variation movement patterns that areunique to each individual, and these fatigue patterns may be a majorcontributor to running related injuries. Because each individualexhibits fatigue in a unique way, a calibration process may be used totrain the machine learning algorithm to identify fatigue for eachindividual. This training may involve sampling movement patterns at thebeginning and ending of a “long” run (relative to the user's fitnesslevel), then classifying these patterns with machine learning. Once thecalibration has been completed, the system can take samples throughoutthe user's run, and in turn, provide feedback indicating fatigue level,as well as provide recommendations on when the user should rest in orderto minimize risk of injury. Such a calibration process is very accurate,because it can capture nuances of movement bio-variations exhibited byone person, which may not be shared by others.

In a particular embodiment, a user may begin by placing an inertialsensor on the sacrum and going for a long run relative to the user'sfitness level sufficient to induce a state of fatigue. Data samples offive strides each may be recorded throughout the run. Samples recordednear the beginning of the run are considered “non-fatigued,” and samplesrecorded near the end of the run are considered “fatigued.” In someembodiments, the inertial sensors may measure both linear acceleration,as well as angular orientation and possibly additional measurements.This higher dimensional data may be processed by first projecting to atwo-dimensional space, and then joining the data-points with a line orcurve in order to generate an image showing the orbital pattern of themovement over five cycles. Data samples from the beginning and end ofthe run may then be used to train a machine learning algorithm toclassify the pattern as either fatigued or non-fatigued.

It should be appreciated that similar techniques may be applied forother movements associated with running or other exercises. For example,in some embodiments, the technologies described herein may be used todetect when an individual is sufficiently “warmed up” that they can nowengage in their run or other exercises. In particular, samples recordedat the beginning of a run (or other exercise) may be considered to be ina “warm up” period. As such, data samples from the beginning and laterportions of the run (or other exercise) may then be used to train amachine learning algorithm to classify the pattern as being associatedwith the individual being either sufficiently warmed up orinsufficiently warmed up for the main run/exercises to be performedsafely and/or effectively.

Fractal analysis may be performed to provide additional insight intochanges in the complexity of the movement pattern due to fatigue. Oncethis calibration process has been completed, the user's unique fatigue“portrait” can be detected and distinguished from the non-fatiguedstate. The patterns may be monitored in real-time and feedback providedto the user. Patterns may also be recorded for later analysis, and mayindicate the percentage of samples indicating fatigue. In addition tothe machine learning analysis, aesthetic images may be generated byoverlaying individual strides and incrementing the pixel value wherestrides overlap to create a “heat-map” type image. This image may beused by itself as visual feedback. Moreover, in some embodiments, anaesthetic orbital pattern is produced, which can then be used to guidemovement and/or share with other users. This feature is particularlyuseful when compared to conventional systems, which rely on time seriesanalyses that may display graphs of a measurement versus time, yet donot generate a unique view of the orbital pattern as formed byrepetitive movements.

The techniques may be further expanded to include the analysis of apatient suffering from a neurodegenerative disease, such as Parkinson'sdisease for instance. For this application, the process proceeds in amanner similar to that described above, with the exception that thefatigued state is initially determined by a trained physical therapistor other trained medical professional in order to calibrate the machinelearning system. The system may be trained to identify fatigued patternsas indicated by a trained observer, after which the patient may thenexercise on his or her own and receive feedback as to when it isappropriate to rest in order to minimize the risk of injury.

Example 3: Providing Feedback for Weight-Lifting and Other Exercises

Inertial sensors may be placed on the user's body and calibrated. Thecalibration procedure in this example, however, may involve a coach,training partner, or physical therapist to observe the movement of theparticular individual and indicate to the system if the movement patternwas performed with good form in order to train the machine learningalgorithm to identify proper movement mechanics. Subsequent repetitionsof the movement after calibration may then be analyzed and classified,with feedback being provided to the user. This application can beextended to include a wide variety of exercises, artistic movements, orathletic movements. Such a calibration process is very accurate, becauseit can capture nuances of movement bio-variations exhibited by oneperson, which may not be shared by others.

In some embodiments, a user being trained to deadlift a barbell may bestudied. To accomplish this, sensors may be placed on the user, forinstance, along the spine, and angular and linear acceleration data maybe recorded in order to monitor lumbar lordosis, thoracic kyphosis,pelvic shift, and other key indicators of proper lifting form. Asindicated above, a calibration process may first be performed wherein aphysical therapist, coach, or training partner observes each repetitionof the lift and indicates to the system if the movement was performedproperly. After a sufficient number of repetitions have been performedin order to train the machine learning algorithm, the user can thenexercise without the physical therapist, coach, or training partner andthe system will provide feedback regarding whether or not the exerciseis being performed with proper form. In this way, the user is able toexercise independently and still receive guidance. An image of therepetitive motion's orbital pattern may be generated providing visualfeedback regarding the movement pattern. Fractal analysis may beperformed in order to quantify the complexity of the movement pattern,and the results may then be used to assess the quality of the motion.

Other Examples

One embodiment is directed to a unique system, components, and methodsfor leveraging machine learning and/or fractal analysis for classifyingmotion. Other embodiments are directed to apparatuses, systems, devices,hardware, methods, and combinations thereof for leveraging machinelearning and/or fractal analysis for classifying motion.

According to an embodiment, a method may include recording, by awearable computing system, sensor data generated by one or more inertialsensors of the wearable computing system, projecting, by the wearablecomputing system, a three-dimensional movement pattern indicated by thesensor data to at least one two-dimensional plane to generate at leastone two-dimensional projection, analyzing, by the wearable computingsystem, the at least one two-dimensional projection using at least onecomputer vision algorithm, and providing, by the wearable computingsystem, feedback to a user of the wearable computing system regardingthe user's movement based on the analysis of the at least onetwo-dimensional projection.

In some embodiments, projecting the three-dimensional movement patternto the at least one two-dimensional plane may include projecting thethree-dimensional movement pattern to multiple two-dimensional planes togenerate corresponding two-dimensional projections.

In some embodiments, the multiple two-dimensional planes may include aside plane, a front plane, and a top plane.

In some embodiments, analyzing the at least one two-dimensionalprojecting may include analyzing the at least one two-dimensionalprojection using at least one computer vision algorithm and at least onemachine learning algorithm.

In some embodiments, the method may further include training the atleast one machine learning algorithm, and projecting thethree-dimensional movement pattern may include projecting thethree-dimensional movement pattern in response to training the at leastone machine learning algorithm.

In some embodiments, providing feedback to the user regarding the user'smovement may include generating an image indicative of the user'smovement based on the at least one two-dimensional projection.

In some embodiments, generating the image may include generating a heatmap indicative of the user's movement.

In some embodiments, analyzing the at least one two-dimensionalprojection may include authenticating artwork generated by the userbased on the user's movement.

In some embodiments, analyzing the at least one two-dimensionalprojection may include determining whether the user's movement isindicative of fatigue.

In some embodiments, analyzing the at least one two-dimensionalprojection may include determining whether the user's movementcorresponds with a proper movement form.

In some embodiments, analyzing the at least one two-dimensionalprojection may include transmitting the at least one two-dimensionalprojection to a remote computing device for analysis.

According to another embodiment, a system may include at least oneinertial sensor and a wearable computing device comprising at least oneprocessor and at least one memory having a plurality of instructionsstored thereon that, in response to execution by the at least oneprocessor, causes the wearable computing device to receive sensor datagenerated by the at least one sensor of the wearable computing device,project a three-dimensional movement pattern indicated by the sensordata to at least one two-dimensional plane to generate at least onetwo-dimensional projection, analyze the at least one two-dimensionalprojection using at least one computer vision algorithm, and providefeedback to a user of the wearable computing device regarding the user'smovement based on the analysis of the at least one two-dimensionalprojection.

In some embodiments, to analyze the at least one two-dimensionalprojecting may include to analyze the at least one two-dimensionalprojection using at least one computer vision algorithm and at least onemachine learning algorithm.

In some embodiments, to provide feedback to the user regarding theuser's movement may include to generate an image indicative of theuser's movement based on the at least one two-dimensional projection.

In some embodiments, to analyze the at least one two-dimensionalprojection may include to authenticate artwork generated by the userbased on the user's movement.

In some embodiments, to analyze the at least one two-dimensionalprojection may include to determine whether the user's movement isindicative of fatigue.

In some embodiments, to analyze the at least one two-dimensionalprojection may include to determine whether the user's movementcorresponds with a proper movement form.

According to yet another embodiment, a wearable computing system mayinclude at least one processor and at least one memory having aplurality of instructions stored thereon that, in response to executionby the at least one processor, causes the wearable computing system toreceive sensor data generated by at least one inertial sensor worn by auser, project a three-dimensional movement pattern indicated by thesensor data to at least one two-dimensional plane to generate at leastone two-dimensional projection, analyze the at least one two-dimensionalprojection using at least one computer vision algorithm, and providefeedback regarding the user's movement based on the analysis of the atleast one two-dimensional projection.

In some embodiments, to analyze the at least one two-dimensionalprojecting may include to analyze the at least one two-dimensionalprojection using at least one computer vision algorithm and at least onemachine learning algorithm.

In some embodiments, to provide feedback regarding the user's movementmay include to generate an image indicative of the user's movement basedon the at least one two-dimensional projection.

What is claimed is:
 1. A method, comprising: sensing sensor datagenerated by one or more inertial sensors, the sensor data being causedby movement of a user, and the sensor data comprising acceleration datain multiple dimensions; recording the sensor data with a wearablecomputing system configured to be worn by the user; projecting, by thewearable computing system, a three-dimensional acceleration patternindicated by the acceleration data to a two-dimensional plane togenerate at least one two-dimensional acceleration projection having afirst acceleration dimension and a second acceleration dimension, wherethe first acceleration dimension is orthogonal to the secondacceleration dimension and both the first and second accelerationdimensions have units of distance per second squared; providing to theuser, by the wearable computing system, visual feedback regarding theuser movement based on the at least one two-dimensional accelerationprojection; wherein the visual feedback comprises at least onetwo-dimensional orbital image showing an orbital pattern of the usermovement in the first and second acceleration dimensions.
 2. The methodof claim 1, wherein the at least one two-dimensional orbital imagecomprises a series of data points computed based on the sensor data, anda curve joining the series of data points.
 3. The method of claim 2,wherein the at least one two-dimensional orbital image comprises aplurality of orbital cycles and the user movement comprises a pluralityof repetitive user movement cycles; each orbital cycle of the pluralityof orbital cycles representing an individual movement cycle of theplurality of repetitive user movement cycles.
 4. The method of claim 3,wherein each orbital cycle represents an individual stride of the user.5. The method of claim 3, wherein the at least one two-dimensionalorbital image comprises a plurality of pixels, each pixel having anassociated pixel value; and wherein for each individual pixel of theplurality of pixels, the associated pixel value is increased each timean individual orbital cycle of the plurality of orbital cycles overlapsthe individual pixel.
 6. The method of claim 1, wherein projecting thethree-dimensional acceleration pattern to the at least onetwo-dimensional plane comprises projecting the three-dimensionalacceleration pattern to multiple two-dimensional planes to generatecorresponding two-dimensional acceleration projections that each have afirst dimension that has units of distance per second squared in a firstdirection and a second dimension that has units of distance per secondsquared in a second direction, where the first direction is orthogonalto the second direction.
 7. The method of claim 6, wherein the multipletwo-dimensional planes comprise a side plane, a front plane, and a topplane.
 8. The method of claim 1, further comprising analyzing the atleast one two-dimensional orbital image to determine whether it isindicative of potential injury or inefficient motion.
 9. The method ofclaim 1, further comprising analyzing the at least one two-dimensionalorbital image to determine whether it corresponds with a proper movementform.
 10. The method of claim 1, further comprising transmitting the atleast one two-dimensional acceleration projection to a remote computingdevice for analysis.
 11. A wearable computing device configured to beworn by a user, the wearable computing device comprising: one or moreinertial sensors configured to generate sensor data based on movement ofthe user, the sensor data comprising acceleration data in multipledimensions; at least one processor configured to: receive the sensordata generated by the one or more inertial sensors; project athree-dimensional acceleration pattern indicated by the accelerationdata to a two-dimensional plane to generate at least one two-dimensionalacceleration projection that has a first acceleration dimension and asecond acceleration dimension, where the first acceleration dimension isorthogonal to the second acceleration dimension and both the first andsecond acceleration dimensions have units of distance per secondsquared; provide visual feedback of the user movement based on the atleast one two-dimensional acceleration projection; wherein the visualfeedback comprises at least one two-dimensional orbital image showing anorbital pattern of the user movement in the first and secondacceleration dimensions.
 12. The wearable computing device of claim 11,wherein the at least one two-dimensional orbital image comprises aseries of data points computed based on the sensor data, and a curvejoining the series of data points.
 13. The wearable computing device ofclaim 12, wherein the at least one two-dimensional orbital imagecomprises a plurality of orbital cycles and the user movement comprisesa plurality of repetitive user movement cycles; where each orbital cycleof the plurality of orbital cycles represents an individual movementcycle of the plurality of repetitive user movement cycles.
 14. Thewearable computing device of claim 13, wherein each orbital cyclerepresents an individual stride of the user.
 15. The wearable computingdevice of claim 13, wherein the at least one two-dimensional orbitalimage comprises a plurality of pixels, each pixel of the plurality ofpixels having an associated pixel value; and wherein for each individualpixel of the plurality of pixels, the associated pixel value isincreased each time an individual orbital cycle of the plurality oforbital cycles overlaps the individual pixel.
 16. The wearable computingdevice of claim 11, wherein the at least one processor is furtherconfigured to analyze the at least one two-dimensional orbital image todetermine whether it is indicative of potential injury or inefficientmotion.
 17. The wearable computing device of claim 11, wherein the atleast one processor is further configured to analyze the at least onetwo-dimensional orbital image to determine whether it corresponds with aproper movement form.
 18. The wearable computing device of claim 1,further comprising a transmitter configured to transmit the at least onetwo-dimensional acceleration projection to a remote computing device foranalysis.
 19. A method, comprising: sensing sensor data generated by oneor more inertial sensors, the sensor data being caused by movement of auser, and the sensor data comprising acceleration data in multipledimensions; recording the sensor data with a wearable computing systemconfigured to be worn by the user; projecting, by the wearable computingsystem, a three-dimensional acceleration pattern indicated by theacceleration data to a two-dimensional plane to generate at least onetwo-dimensional acceleration projection having a first accelerationdimension and a second acceleration dimension, where the firstacceleration dimension is orthogonal to the second accelerationdimension and both the first and second acceleration dimensions haveunits of distance per second squared; analyzing the at least onetwo-dimensional acceleration projection using fractal analysis atmultiple scales and taking into account variations in fractal dimensionsat different scales: providing to the user, by the wearable computingsystem, visual feedback regarding the user movement based on the atleast one two-dimensional acceleration projection; wherein the visualfeedback comprises at least one two-dimensional orbital image showing anorbital pattern of the user movement in the first and secondacceleration dimensions.
 20. The method of claim 19, wherein the atleast one two-dimensional orbital image comprises a plurality of orbitalcycles and the user movement comprises a plurality of repetitive usermovement cycles; each orbital cycle of the plurality of orbital cyclesrepresenting an individual movement cycle of the plurality of repetitiveuser movement cycles.